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Highlights 


Why  GAO  Did  This  Study 


What  GAO  Found 


Federal  agencies  with  a  budget  of  at 
least  $100  million  for  research  and 
development  (R&D)  conducted  by 
others  must  participate  in  the  Small 
Business  Innovation  Research  (SBIR) 
program.  SBIR  has  four  purposes: 
meet  federal  R&D  needs;  stimulate 
technological  innovation;  increase 
commercialization  (e.g.,  sales)  of 
innovations  based  on  federal  R&D;  and 
encourage  participation  in  innovation 
by  small  businesses  owned  by 
disadvantaged  individuals  and  women. 
The  Small  Business  Administration 
(SBA)  oversees  the  efforts  of 
participating  agencies,  which  make 
awards  to  small  businesses  using 
SBIR  funds.  Congress  directed  SBA  to 
develop  a  database  with 
commercialization  data  for  government 
use  in  evaluating  the  program.  GAO 
was  asked  to  determine  (1)  how 
agencies  have  addressed  SBIR’s 
purposes  and  (2)  the  extent  of  data 
available  to  evaluate  progress  in 
increasing  commercialization.  GAO 
analyzed  program  documents  and 
interviewed  officials  at  SBA  and  five 
agencies  that  accounted  for  about 
96  percent  of  SBIR  awards. 

What  GAO  Recommends 

GAO  recommends  that  SBA  work  with 
participating  agencies  to  (1)  collect 
data  on  applications  from  small 
businesses  owned  by  disadvantaged 
individuals  and  women  and  (2)  identify 
best  practices  for  verification  of 
commercialization  data.  SBA,  DOE, 
and  NSF  generally  agreed  with  these 
recommendations;  the  other  agencies 
GAO  reviewed  neither  agreed  nor 
disagreed. 


For  fiscal  years  2008  through  2011,  the  participating  agencies  GAO  reviewed — 
the  Department  of  Defense  (DOD),  the  Department  of  Energy  (DOE),  the 
National  Aeronautics  and  Space  Administration,  the  Department  of  Health  and 
Human  Services’  National  Institutes  of  Health,  and  the  National  Science 
Foundation  (NSF) — addressed  SBIR’s  purposes  through  solicitations  for  award 
applications,  technical  assistance  or  matching  funds  programs,  and  outreach.  In 
particular,  the  agencies  addressed  SBIR  purposes  related  to  meeting  federal 
R&D  needs  and  stimulating  technological  innovation  through  their  solicitations, 
which  included  research  topics  that  were  designed  to  meet  agencies’  respective 
R&D  or  mission  needs.  Agencies  also  addressed  commercialization  of 
innovations  through  solicitations,  as  well  as  through  technical  assistance  for 
award  recipients  or  through  matching  funds  programs.  To  provide  technical 
assistance,  the  agencies  contracted  with  vendors  and  consultants  for  help  in 
developing  business  plans  and  identifying  potential  customers  for  SBIR  award 
recipients,  among  other  things.  Agency  matching  funds  programs  provided 
additional  SBIR  funds  to  award  recipients  that  obtained  commitments  from 
outside  investors.  Agencies  generally  addressed  the  remaining  SBIR  purpose, 
encouraging  participation  by  small  businesses  owned  by  disadvantaged 
individuals  and  women,  through  outreach  activities  aimed  at  a  broader  audience, 
such  as  sharing  information  on  Web  sites.  However,  the  effectiveness  of  these 
efforts  is  difficult  to  evaluate,  in  part  because  SBA  does  not  collect  data  on  the 
number  of  SBIR  applications  submitted  by  such  businesses,  thus  hindering 
analysis  of  trends  in  their  submission  of  applications. 

Comparable  data  are  not  available  across  participating  agencies  to  evaluate 
progress  in  increasing  commercialization  of  SBIR  technologies.  SBA  has  not  yet 
expanded  an  existing  database  to  include  commercialization  data  for  program 
evaluation,  but  the  agency  has  hired  a  contractor  and  allocated  funds  to  develop 
the  expanded  database  by  August  201 1 .  SBA  has  also  worked  with  participating 
agencies  to  develop  common  metrics  for  commercialization.  In  the  absence  of 
the  expanded  database,  agencies  have  independently  gathered 
commercialization  data  for  their  own  use  that  are  not  comparable.  In  collecting 
these  data,  agencies  differed  in  the  types  of  data  collection  instruments  used, 
dates  the  instruments  were  administered,  award  recipient  populations  queried, 
and  types  of  data  requested.  Furthermore,  with  the  exception  of  DOD,  agencies 
that  GAO  reviewed  did  not  generally  take  steps  to  verify  commercialization  data 
they  collected  from  award  recipients,  so  the  accuracy  of  the  data  is  largely 
unknown.  SBA  has  worked  with  SBIR  agencies  to  identify  best  practices  in  other 
areas  of  program  management  but  has  not  identified  best  practices  for  agencies 
to  use  in  verifying  the  accuracy  of  commercialization  data.  Implementing  the 
expanded  database  should  improve  the  comparability  of  commercialization  data 
available,  but  a  lack  of  consistent  practices  for  verifying  the  accuracy  of  these 
data  may  limit  their  usefulness  for  programwide  evaluation. 
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Abbreviations 


DOD 

Department  of  Defense 

DOE 

Department  of  Energy 

NASA 

National  Aeronautics  and  Space  Administration 

NIH 

National  Institutes  of  Health 

NSF 

National  Science  Foundation 

R&D 

Research  and  Development 

Recovery  Act 

American  Recovery  and  Reinvestment  Act 

SBA 

Small  Business  Administration 

SBIR 

Small  Business  Innovation  Research 

STTR 

Small  Business  Technology  Transfer 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


August  15,  201 1 

The  Honorable  Ben  Quayle 
Chairman 

Subcommittee  on  Technology 
and  Innovation 

Committee  on  Science,  Space, 
and  Technology 
House  of  Representatives 

The  Honorable  Adrian  Smith 
House  of  Representatives 

Small  businesses  have  been  collectively  described  as  an  engine  for 
economic  growth  and  innovation.  According  to  the  Small  Business 
Administration  (SBA),  such  businesses  have  been  responsible  for 
creating  64  percent  of  net  new  jobs  over  the  past  1 5  years  and  have 
produced  13  times  more  patents  per  employee  than  larger  firms.  To 
assist  small  businesses  in  undertaking  and  obtaining  the  benefits  of 
research  and  development  (R&D),  Congress  first  passed  legislation 
authorizing  the  Small  Business  Innovation  Research  (SBIR)  program  in 
1982.  SBA’s  Office  of  Technology  administers  the  program,  which 
presently  has  four  overarching  purposes:  to  use  small  businesses  to  meet 
federal  R&D  needs,  stimulate  technological  innovation,  increase 
commercialization  of  innovations  derived  from  federal  R&D  efforts,1  and 
encourage  participation  in  technological  innovation  by  small  businesses 


Vor  the  SBIR  program,  SBA  has  defined  commercialization  as  the  “process  of  developing 
marketable  products  or  services  and  producing  and  delivering  products  or  services  for 
sale  (whether  by  the  originating  party  or  by  others)  to  Government  or  commercial 
markets.”  Small  Business  Innovation  Research  Program  Policy  Directive,  67  Fed.  Reg. 
60,072,  60,083  (Sept.  24,  2002). 
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owned  by  disadvantaged  individuals2  and  women.3  Since  the  program 
was  first  authorized  in  1982,  there  has  been  increased  congressional 
interest  in  its  results,  particularly  the  commercialization  aspect. 

Every  federal  agency  with  a  budget  of  $100  million  or  more  for  extramural 
R&D — which  is  conducted  by  nonfederal  employees  outside  federal 
facilities — is  required  to  establish  and  operate  an  SBIR  program  funded 
by  2.5  percent  of  that  budget.  Eleven  agencies  currently  participate  in  the 
program.4  In  fiscal  year  2009  (the  most  recent  year  for  which  SBA  data 
are  available),  the  11  agencies  reported  more  than  $2.2  billion  in  SBIR 
awards.  Although  each  of  these  agencies  manages  its  own  program,  SBA 
oversees  and  coordinates  agency  efforts  by  setting  policy,  collecting 
program  data,  reviewing  agency  progress,  and  reporting  annually  to 
Congress. 

Since  the  creation  of  the  SBIR  program  29  years  ago,  it  has  been 
reauthorized,  modified,  and  extended  by  Congress  at  various  times.  The 
Small  Business  Reauthorization  Act  of  2000  extended  the  program 
through  September  30,  2008.  This  act  was  succeeded  by  a  series  of 


Generally,  socially  disadvantaged  individuals  are  those  who  have  been  subjected  to 
racial  or  ethnic  prejudice  or  cultural  bias  within  American  society  because  of  their  identities 
as  members  of  groups  and  without  regard  to  their  individual  qualities.  Economically 
disadvantaged  individuals  are  socially  disadvantaged  individuals  whose  ability  to  compete 
in  the  free  enterprise  system  has  been  impaired  by  diminished  capital  and  credit 
opportunities  as  compared  with  that  of  others  in  the  same  or  similar  line  of  business  who 
are  not  socially  disadvantaged.  Throughout  this  report,  we  refer  to  socially  and 
economically  disadvantaged  individuals  as  “disadvantaged  individuals.”  To  be  considered 
owned  by  disadvantaged  individuals,  businesses  must  be  at  least  51  percent  owned  by 
one  or  more  disadvantaged  individuals  or,  in  the  case  of  any  publicly  owned  business,  at 
least  51  percent  of  the  stock  must  be  owned  by  disadvantaged  individuals,  and  those  who 
are  disadvantaged  must  control  the  management  and  daily  business  operations. 

3Encouraging  participation  by  women-owned  businesses  was  not  included  among  the 
program’s  original  purposes  but  was  added  by  the  Small  Business  Research  and 
Development  Enhancement  Act  of  1 992.  Pub.  L.  No.  1 02-564,  §  1 02,  1 06  Stat.  4249, 

4250.  To  be  women-owned,  a  small  business  must  be  at  least  51  percent  owned  by  one 
or  more  women  or,  in  the  case  of  any  publicly  owned  business,  at  least  51  percent  of  the 
stock  must  be  owned  by  women,  and  women  must  control  the  management  and  daily 
business  operations. 

4The  1 1  agencies  are  the  U.S.  Department  of  Agriculture,  Department  of  Commerce, 
Department  of  Defense,  Department  of  Education,  Department  of  Energy,  Department  of 
Health  and  Human  Services,  Department  of  Homeland  Security,  Department  of 
Transportation,  Environmental  Protection  Agency,  National  Aeronautics  and  Space 
Administration,  and  National  Science  Foundation. 
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temporary  extensions,  the  most  recent  of  which  extended  the  program 
through  September  30,  201 1 . 

In  this  context,  you  asked  us  to  examine  the  practices  of  participating 
agencies.  Accordingly,  we  agreed  to  determine  (1)  how  participating 
agencies  have  addressed  the  SBIR  program’s  four  overarching  purposes 
when  implementing  their  programs  and  (2)  the  extent  of  SBIR  program 
data  available  to  evaluate  progress  in  increasing  commercialization  of 
SBIR  technologies. 

Our  review  of  the  program  focused  on  SBA  and  five  agencies  that 
accounted  for  about  96  percent  of  the  total  dollars  awarded  by  the 
program  in  2009:  the  Department  of  Defense  (DOD),  the  Department  of 
Energy  (DOE),  the  National  Aeronautics  and  Space  Administration 
(NASA),  the  Department  of  Health  and  Human  Services’  National 
Institutes  of  Health  (NIH),5  and  the  National  Science  Foundation  (NSF).6 
At  two  of  these  agencies — DOD  and  NIH — we  also  reviewed  program 
activities  conducted  by  the  three  subcomponent  agencies  with  the  largest 
SBIR  budgets  because  some  key  activities  are  carried  out  at  that  level. 
Specifically,  for  DOD,  we  examined  the  SBIR  programs  of  the  Army,  Air 
Force,  and  Navy  and  for  NIH,  we  examined  the  programs  of  the  National 
Institute  of  Allergy  and  Infectious  Diseases;  the  National  Cancer  Institute; 
and  the  National  Heart,  Lung,  and  Blood  Institute.  In  conducting  our 
review,  we  analyzed  documentation  from  these  agencies  for  fiscal  years 
2008  through  201 0  and,  when  possible,  for  fiscal  year  201 1 .  The 
documentation  we  reviewed  included,  among  other  things,  policy 
guidance,  solicitations  for  applications  for  SBIR  awards,  and  descriptions 
of  commercialization  assistance  provided  to  SBIR  awardees.  We  also 
reviewed  applicable  laws  and  regulations.  In  addition,  we  interviewed 
SBIR  program  officials  at  the  agencies,  as  well  as  inspector  general  staff 
at  NSF  and  knowledgeable  stakeholders,  including  staff  of  the  National 
Academy  of  Sciences’  National  Research  Council  and  representatives  of 


5The  National  Institutes  of  Health  accounted  for  more  than  98  percent  of  the  SBIR 
expenditures  of  the  Department  of  Health  and  Human  Services  from  fiscal  year  2000 
through  fiscal  year  2010.  Four  other  agencies  in  the  department — the  Centers  for 
Medicaid  and  Medicaid  Services,  the  Centers  for  Disease  Control  and  Prevention,  the 
Administration  for  Children  and  Families,  and  the  Food  and  Drug  Administration — also 
issued  SBIR  solicitations  during  that  period. 

6While  the  agencies  that  we  reviewed  account  for  the  vast  majority  of  SBIR  program 
expenditures,  the  results  of  our  review  cannot  be  generalized  to  all  participating  agencies. 
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trade  associations.  We  selected  the  trade  associations  based  on  their 
familiarity  with  the  program,  the  technologies  on  which  they  focus,  and 
whether  their  membership  includes  small  businesses  owned  by 
disadvantaged  individuals  and  women.7  To  obtain  further  context  for  our 
review,  we  attended  two  national  conferences  on  the  SBIR  program,  as 
well  as  a  workshop  conducted  by  the  National  Research  Council.  We 
conducted  this  performance  audit  from  June  201 0  to  August  201 1 ,  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  we  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  A  more  detailed 
description  of  the  scope  of  our  review  and  the  methods  we  used  is 
contained  in  appendix  I. 


Background 


SBA  coordinates  and  oversees  the  efforts  of  the  1 1  agencies  currently 
participating  in  the  SBIR  program.  SBA  coordinates  the  agencies’ 
schedules  for  issuing  solicitations — announcements  of  opportunities  for 
small  businesses  to  apply  for  awards — and  provides  access  to  these 
solicitations  through  its  Web  site.  As  part  of  its  oversight  effort,  SBA 
collects  SBIR  data  from  the  participating  agencies,  aggregates  the  data, 
and  uses  the  data  to,  among  other  things,  monitor  the  program  and  report 
to  Congress.  SBA  also  provides  guidance  to  participating  agencies  on  the 
general  conduct  and  operation  of  the  program,  which  it  periodically 
updates,  for  example,  in  response  to  changes  in  the  program’s 
authorizing  legislation.  Under  the  legislation  and  SBA’s  guidance, 
agencies  have  considerable  flexibility  to  design  their  programs.  For 
example,  each  agency  determines,  in  consultation  with  SBA,  such  items 
as  the  number  of  solicitations  to  be  issued  during  a  fiscal  year  and  the 
dates  applications  are  due.  Agencies  also  have  discretion  to  determine 
what  type  of  research  to  include  in  their  solicitations,  how  to  review 
applications  for  technical  and  scientific  merit,  which  applications  to  fund, 
and  the  size  of  the  award,  among  other  things. 


7The  views  expressed  by  representatives  of  these  trade  associations  cannot  be 
generalized  to  other  trade  associations. 
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The  Small  Business  Innovation  Development  Act  of  1982  provided  for  a 
competitive  three-phased  SBIR  program.  In  phase  I,  participating 
agencies  award  up  to  $150,000  for  a  period  of  about  6  to  9  months  to 
small  businesses  to  conduct  experimental  or  theoretical  R&D.  Small 
businesses  whose  phase  I  projects  demonstrate  scientific  and  technical 
merit,  in  addition  to  commercial  potential,  may  compete  for  phase  II 
awards  of  up  to  $1  million  to  continue  the  R&D  for  an  additional  period, 
normally  not  to  exceed  2  years.8  Phase  I  and  II  award  funds  may  be  used 
for  costs  related  to  conducting  the  research,  such  as  salaries,  fringe 
benefits,  equipment,  and  consulting  services,  as  well  as  for  profits  and 
fees.  To  be  eligible  for  a  phase  I  or  II  SBIR  award,  a  business  must  have 
500  or  fewer  employees,  be  organized  for  profit  with  a  place  of  business 
in  the  United  States,  and  operate  primarily  in  the  United  States  or  make  a 
significant  contribution  to  the  U.S.  economy.  Generally,  a  business  must 
also  be  at  least  51  percent  owned  and  controlled  by  one  or  more 
individuals  who  are  U.S.  citizens  or  permanent  resident  aliens.9  These 
eligibility  requirements  apply  at  the  time  that  a  phase  I  or  II  award  is 
made.  During  phase  III,  businesses  must  secure  non-SBIR  funding  to 
develop  the  commercial  potential  of  the  innovative  technologies  resulting 
from  their  SBIR  projects;10  such  funding  may  come  from  the  private 
sector,  federal  agencies,  or  other  sources. 

As  the  program  has  been  reauthorized  over  the  years,  legislation  has 
established  a  number  of  requirements  related  to  the  program’s  purposes. 
For  example,  the  Small  Business  Research  and  Development 
Enhancement  Act  of  1992  directed  SBA  to  make  more  information 
available  about  the  SBIR  program,  particularly  about  participation  by 
small  businesses  owned  by  disadvantaged  individuals  and  women,  and 
required  that  agencies  increase  their  outreach  to  such  businesses.  In 
addition,  the  Small  Business  Reauthorization  Act  of  2000  directed  that 
applicants  for  phase  II  SBIR  awards  be  required  to  submit 


8Priorto  2010,  the  award  ceiling  for  phase  I  was  $100,000,  and  the  award  ceiling  for 
phase  II  was  $750,000.  According  to  SBA  guidance,  agencies  may  exceed  these  ceilings 
with  appropriate  justification. 

9ln  addition,  small  businesses  that  are  majority-owned  or  -controlled  by  another  business 
or  joint  venture  that  meets  the  51  percent  requirement  are  also  eligible  to  compete  for 
SBIR  awards.  13  C.F.R.  §  121.702(a). 

i  n 

In  this  report,  we  are  using  “technologies”  to  refer  to  products,  systems,  methods, 
services  or  other  results  of  SBIR-funded  R&D. 
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commercialization  plans,11  and  it  mandated  that  SBA  develop,  maintain, 
and  make  available  to  the  public  a  database  that  contained  SBIR  award 
data.  In  addition,  the  act  required  SBA  to  develop  and  maintain,  by  June 
2001,  a  restricted  government-use  database  that  would  contain  award- 
related  data  from  the  public  database,  as  well  as  additional  confidential 
data  that  would  be  accessible  only  to  government  agencies  and  other 
authorized  users.  The  act  stated  that  this  database  would  be  used 
exclusively  for  program  evaluation — which,  as  we  have  noted  in  past 
work,  involves  the  systematic  collection  and  analysis  of  accurate, 
comparable,  and  complete  data  on  program  results.12  The  act  required 
the  government-use  database  to  contain,  among  other  things,  data  that 
applicants  for  phase  II  awards  would  be  required  to  supply  on  the 
commercialization  success  of  any  prior  phase  II  awards,  such  as  data  on 
sales  of  or  additional  investment  in  the  technologies  funded  under  the 
awards.  The  act  further  specified  that  the  government-use  database 
would  contain  annual  updates  to  these  data,  which  phase  II  award 
recipients  would  be  requested  to  voluntarily  provide  for  5  years  after  the 
period  covered  by  the  award. 

To  accomplish  this  mandate,  SBA  envisioned  expanding  an  electronic 
database,  known  as  Tech-Net,  which  it  had  developed  in  the  late  1990s, 
into  two  sections:  a  public-use  portion  and  a  government-use  portion 
containing  commercialization  data.  The  public-use  portion  of  the  database 
has  been  available  since  2000,  according  to  SBA,  and  it  contains  such 
award-related  data  as  the  phase  of  the  award,  amount  of  the  award, 
name  and  location  of  the  business  receiving  the  award,  an  abstract  of  the 
work  to  be  conducted  under  the  award,  and  whether  the  business  is 
categorized  as  owned  by  disadvantaged  individuals  or  women.  In  October 
2006,  however,  we  reported  that  some  SBIR  agencies  did  not 
consistently  provide  or  correctly  format  the  awards-related  data  for 
several  fields  in  the  public-use  portion  of  the  database.13  For  example, 


^The  requirement  specifically  applies  to  technologies  that  are  moving  toward 
commercialization. 

12See  GAO,  Small  Business  Innovation  Research:  Agencies  Need  to  Strengthen  Efforts  to 
Improve  the  Completeness,  Consistency,  and  Accuracy  of  Awards  Data,  GAO-07-38 
(Washington,  D.C.:  Oct.  19,  2006)  and  GAO,  Managing  for  Results:  Measuring  Program 
Results  That  Are  under  Limited  Federal  Control,  GGD-99-16 
(Washington,  D.C.:  Dec.  11,  1998). 

13GAO-07-38. 
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two  of  the  eight  agencies  we  reviewed  had  not  consistently  provided  data 
on  whether  the  businesses  receiving  the  awards  were  categorized  as 
owned  by  disadvantaged  individuals  or  women.14  At  that  time,  we  also 
reported  that  SBA  had  not  implemented  the  government-use  portion  of 
the  database,  primarily,  according  to  SBA  officials,  because  of  increased 
security  requirements  for  the  database,  agency  management  changes, 
and  budgetary  constraints.  Additionally,  we  reported  that  while  five  of  the 
agencies  we  reviewed  had  systematically  collected  commercialization 
data,  their  data  collection  efforts  differed  in  ways  that  made  it  challenging 
to  evaluate  the  program  across  agencies.  In  August  2009,  we  testified 
before  Congress  that  SBA  said  the  database  would  no  longer  accept 
incorrectly  formatted  awards-related  data  from  participating  agencies.15 

A  committee  of  the  National  Academy  of  Sciences’  National  Research 
Council  has  conducted  a  series  of  assessments  of  the  SBIR  program, 
both  within  and  across  agencies,  as  part  of  a  legislatively  mandated 
study.  The  results  were  summarized  in  a  single  report,  in  which  the 
committee  stated  that  SBIR  is  making  significant  progress  in  achieving 
congressional  goals.16  The  study  concluded  that  the  SBIR  program  is 
“sound  in  concept  and  effective  in  practice.”  The  study  also 
recommended  changes  that  could  make  the  program  more  effective. 
Among  other  things,  the  study  recommended  that  SBA  and  participating 
agencies  improve  the  collection  of  data  that  track  participation  in  the  SBIR 
program  by  businesses  owned  by  disadvantaged  individuals  and  women, 
develop  targeted  outreach  to  such  businesses  that  is  based  on  an 
analysis  of  factors  that  affect  their  participation,  and  improve 
documentation  of  commercialization  success.  The  National  Research 
Council  is  now  undertaking  another  round  of  assessments  to  provide  a 
second  snapshot  of  the  program’s  progress  in  achieving  its  legislative 
purposes. 


14For  our  2006  report,  we  reviewed  the  SBIR  programs  of  the  Department  of  Agriculture, 
Environmental  Protection  Agency,  and  National  Institute  of  Standards  and  Technology  in 
addition  to  those  programs  of  the  agencies  we  reviewed  for  this  report. 

15GAO,  Small  Business  Innovation  Research:  Observations  on  Agencies’  Data  Collection 
and  Eligibility  Determination  Efforts,  GAO-09-956T  (Washington,  D.C.:  Aug.  6,  2009). 

16National  Research  Council,  An  Assessment  of  the  SBIR  Program  (Washington,  D.C.: 
National  Academies  Press,  2008).  This  report  is  part  of  a  series  published  by  the  National 
Academies  in  response  to  the  mandate. 
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Agencies  Addressed 
SBIR’s  Purposes 
through  Solicitations, 
Technical  Assistance, 
Matching  Funds 
Programs,  and 
Outreach 


For  fiscal  years  2008  through  201 1 ,  the  five  participating  agencies  we 
reviewed  addressed  the  SBIR  purposes  of  using  small  business  to  meet 
federal  R&D  needs  and  stimulating  technological  innovation  through  their 
solicitations.  Agencies  also  used  solicitations,  as  well  as  technical 
assistance  or  matching  funds  programs,  to  address  the  SBIR  purpose  of 
increasing  commercialization  of  innovations  derived  from  federal  R&D 
efforts.  To  address  the  remaining  program  purpose — encouraging 
participation  in  technological  innovation  by  small  businesses  owned  by 
disadvantaged  individuals  and  women — agencies  relied  mainly  on 
outreach  activities  aimed  at  a  broader  audience. 


Agencies  Addressed  the 
SBIR  Purposes  Related  to 
Meeting  Federal  Research 
Needs  and  Stimulating 
Innovation  through  Their 
Solicitations 


All  of  the  participating  agencies  that  we  reviewed  designed  the  SBIR 
solicitations  that  they  issued  for  fiscal  years  2008  through  201 1  to  meet 
federal  R&D  or  mission  needs  and  stimulate  technological  innovation.17 
All  of  these  agencies  selected  research  topics  for  their  solicitations  that 
were  designed  to  meet  their  respective  R&D  or  mission  needs  and 
specified  that  applications  would  be  evaluated  on  the  basis  of 
responsiveness  to  those  topics.  The  agencies  that  purchase  SBIR-funded 
technologies  for  their  own  use — DOD,  DOE,  and  NASA — tended  to  select 
solicitation  topics  that  met  specific  agency  needs  for  R&D.  For  example, 
in  fiscal  year  201 1 ,  DOD  solicited  applications  to  develop  a  fuel  cell 
system  capable  of  converting  ethanol  into  electricity  in  an  efficient,  small, 
lightweight,  portable  power  system.  According  to  the  solicitation,  such 
advanced  fuel  cell  systems  could  provide  soldiers  power  to  complement 
batteries  and  to  charge  rechargeable  batteries,  reducing  the  number  of 
batteries  required  for  extended  time  in  the  field.  In  contrast,  NIH  and  NSF, 
which  generally  do  not  purchase  SBIR-funded  technologies,  tended  to 
issue  solicitations  for  a  broader  spectrum  of  R&D  to  support  their 
missions  of  advancing  biomedical  and  other  scientific  and  engineering 
disciplines.  Among  the  agencies  we  reviewed,  NIH  and  its  components 
gave  applicants  the  most  leeway  in  addressing  agency  needs:  rather  than 
limiting  applications  to  specific  research  topics  identified  in  solicitations, 
NIH  and  its  components  usually  listed  suggested  topics  and  encouraged 
applicants  to  propose  innovative  projects  that  fit  the  agency’s  mission. 


17Our  review  did  not  include  NASA’s  2011  SBIR  solicitation,  which  was  released  in  July 
2011. 
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Concerning  innovation,  each  of  the  agencies  included  instructions  in  its 
SBIR  solicitations  about  the  type  of  information  applicants  had  to  provide 
about  the  innovativeness  of  the  proposed  work.  For  example,  NASA 
informed  phase  I  and  II  applicants  that  a  competitive  application  would 
describe  the  proposed  innovation  relative  to  state-of-the-art  knowledge  in 
the  field,  among  other  things.  In  addition,  these  agencies  explained  to 
applicants  how  reviewers  would  consider  evidence  of  the  innovativeness 
of  the  applicants’  proposed  research  approaches.  For  example,  in  its 
fiscal  year  2010  solicitation,  NSF  stated  that  applications  would  be 
evaluated,  in  part,  on  the  basis  of  whether  they  reflected  state-of-the-art 
knowledge  in  the  major  research  activities  proposed  and  whether  the 
work  was  likely  to  advance  state-of-the-art  knowledge. 


Agencies  Worked  to 
Increase 

Commercialization  of 
Innovations  through 
Solicitations,  Technical 
Assistance,  and  Matching 
Funds  Programs 


The  participating  agencies  we  reviewed  addressed  the  SBIR  purpose  of 
increasing  commercialization  of  innovations  through  solicitations,  as  well 
as  through  technical  assistance  or  matching  funds  programs. 

Solicitations.  Of  the  five  agencies  we  reviewed,  all  but  NIH  required  in 
their  solicitations  for  fiscal  years  2008  through  201 1  that  applicants  for 
phase  I  awards  submit  a  commercialization  strategy  demonstrating  that 
the  applicants  had  taken  steps  such  as  identifying  a  market  for  their  SBIR 
technologies,  planning  to  secure  financing,  and  estimating  expected 
future  sales.  For  phase  II  awards,  all  of  the  agencies  we  reviewed 
required  that  applicants  submit  a  commercialization  plan.18  In  general,  the 
solicitations  we  reviewed  required  that  phase  II  commercialization  plans 
discuss  the  potential  market  and  competitors;  the  qualifications  of  key 
management  and  technical  personnel;  as  well  as  financing,  marketing, 
and  manufacturing  plans,  among  other  things. 


The  agencies  we  reviewed  differed  in  their  stated  processes  for 
evaluating  the  commercial  potential  of  applications.  For  example,  DOD 
guidance  to  applicants  outlined  a  systematic  process  for  how  the  agency 
would  consider  commercialization  potential  when  evaluating  applications 
submitted  by  small  businesses  that  had  received  multiple  prior  awards. 
DOD  indicated  that,  under  this  process,  it  would  assign  a 
commercialization  achievement  score  to  applicants  that  had  completed 


1  ft 

NSF  does  not  issue  separate  solicitations  for  phase  II  projects  and  considers  phase  II 
proposals  only  from  recipients  of  NSF  phase  I  awards. 
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the  work  for  four  or  more  phase  II  awards  from  any  agency;19  this  score 
would  reflect  how  the  applicants’  commercialization  experience  compared 
with  historical  averages.  Applicants  whose  scores  fell  within  the  lowest  20 
percent  would  not  be  allowed  to  receive  more  than  half  the  maximum 
number  of  points  possible  for  commercialization  potential,  which  was  to 
be  assessed  on  the  basis  of  several  factors,  including  the 
commercialization  strategy  or  plan.  DOD  guidance  stated  that  businesses 
with  fewer  than  four  completed  phase  II  awards  would  not  be  affected  by 
the  absence  of  a  commercialization  achievement  score.  Although  the 
other  four  agencies  we  reviewed  did  not  outline  as  systematic  a  process 
for  evaluating  past  commercialization  success  as  a  gauge  of 
commercialization  potential,  they  still  indicated  that  commercialization 
potential  would  be  taken  into  account  in  reviewing  applications.  For 
example,  DOE’s  solicitation  instructions  encouraged  phase  I  applicants  to 
seek  firm  commitments  for  private-sector  or  non-SBIR  federal  funding 
prior  to  applying  for  a  phase  II  award.  The  instructions  further  stated  that 
phase  II  applicants  that  obtained  such  commitments  were  more  likely  to 
receive  full  credit  for  commercialization  planning  during  the  evaluation  of 
their  applications.  In  the  case  of  NSF,  solicitation  instructions  stated  that 
proposals  are  usually  reviewed  by  3  to  10  outside  experts  in  fields  related 
to  the  proposal;  according  to  NSF  officials,  these  reviewers  have 
business  experience.  NSF’s  solicitation  instructions  further  stated  that  the 
agency  would  not  review  applications  that  lacked  sufficient  information  on 
commercial  potential. 

In  2010,  two  agencies  we  reviewed  also  issued  SBIR  solicitations  under 
new  programs  that  were  explicitly  oriented  toward  increasing 
commercialization.  Specifically,  in  July  2010,  DOE  launched  a  program 
under  which  it  solicited  applications  for  phase  III  of  SBIR,  the 
commercialization  phase.  DOE  documents  indicated  that  the  agency 
would  make  available  approximately  $30  million,  including  funding  from 
the  American  Recovery  and  Reinvestment  Act  (Recovery  Act),20  for 
phase  III  awards,  which  are  intended  to  allow  businesses  to  pursue 
commercial  applications  of  work  performed  under  phase  I  and  II  awards. 
In  addition,  NIH’s  National  Cancer  Institute  began  a  program  under  which 
it  solicited  phase  I  applications  to  continue  development  of  technologies 


1 Q 

DOD  officials  told  us  that  for  fiscal  years  2008-2010,  such  scores  were  calculated  for 
42-45  percent  of  applicants. 

20American  Recovery  and  Reinvestment  Act  of  2009,  Pub.  L.  No.  111-5,  123  Stat.  115. 
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that  have  originated  in  its  laboratories,  with  the  goal  of  advancing  these 
technologies  toward  commercial  products.21  SBA  has  designated  the  use 
of  the  SBIR  program  to  encourage  commercialization  of  agencies’  internal 
research  as  a  best  practice  on  its  SBIR  Web  site. 

Technical  assistance.  All  five  agencies  included  in  our  review  provided 
technical  assistance  to  help  award  recipients  build  their  capacity  to 
commercialize  their  technologies.  To  provide  the  assistance,  the  agencies 
contracted  with  vendors  and  consultants  who  have  experience  in  bringing 
technologies  to  market.  With  the  exception  of  NASA,  the  agencies 
supported  the  technical  assistance  at  least  in  part  through  the  use  of 
SBIR  funds.22 

In  fiscal  years  2008  to  2010,  DOD,  DOE,  NIH,  and  NSF  spent  SBIR  funds 
on  technical  assistance  for  phase  I  award  recipients.  Some  of  the 
assistance  was  in  the  form  of  interactive  training  Webinars  or  online  tools 
directed  toward  a  broad  spectrum  of  SBIR  applicants  and  award 
recipients.  For  example,  the  Navy  offered  phase  I  award  recipients  the 
use  of  a  software  tool,  known  as  WebTRIMS,  that  helps  identify,  quantify, 
and  track  risks  associated  with  SBIR  technology  development  and  covers 
topics  such  as  contracting  strategies,  business  and  transition  planning, 
and  manufacturing  readiness.  Other  phase  I  assistance  was  more 
customized.  For  example,  DOE  offered  phase  I  award  recipients 
customized  technical  assistance  designed  to  help  them  develop  a 
commercialization  plan  complete  with  an  implementation  schedule  and 
suggestions  for  product  design.  Similarly,  on  a  first-come,  first-served 
basis,  NIH  offered  phase  I  award  recipients  assessments  of  their  SBIR- 
funded  technologies’  likely  niche  in  the  existing  commercial  market,  which 
could  help  recipients  develop  commercialization  plans  for  phase  II 
applications.23  Additionally,  NSF  offered  phase  I  award  recipients 
personalized  mentoring  and  coaching  sessions  with  an  advisor. 


21According  to  NIH  officials,  the  National  Cancer  Institute  based  its  program  on  one 
originated  by  the  National  Institute  of  Standards  and  Technology. 

^Participating  agencies  have  authority  to  spend  or  authorize  expenditure  of  a  portion  of 
their  SBIR  funds  for  technical  assistance  to  SBIR  awardees.  The  value  of  the  technical 
assistance  provided  with  or  authorized  from  SBIR  funds  can  be  up  to  $4,000  per  award  in 
phase  I  and  up  to  $4,000  per  award  per  year  in  phase  II. 

po 

According  to  NIH  officials,  since  the  agency  began  offering  such  assistance  in  2004,  NIH 
has  provided  the  assistance  to  nearly  700  of  the  5,090  businesses  that  received  phase  I 
SBIR  awards. 
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According  to  NSF  officials,  92  percent  of  phase  I  recipients  chose  to 
participate  in  the  technical  assistance  program  in  2010.  NASA  did  not 
provide  technical  assistance  for  phase  I  award  recipients;  NASA  officials 
told  us  they  believed  technical  assistance  would  have  the  most  utility  for 
phase  II  NASA  award  recipients. 

During  at  least  a  portion  of  the  period  we  reviewed,  all  five  agencies 
offered  individualized  technical  assistance  for  phase  II  award  recipients, 
although  DOE  curtailed  such  assistance  in  2010,  and  NASA  discontinued 
its  assistance  in  2008.  Award  recipients  were  selected  for  assistance  on 
the  basis  of  factors  such  as  recommendations  from  SBIR  program  staff 
and  the  award  recipients’  potential  for  rapidly  moving  their  technologies  to 
phase  III.  The  assistance  consisted  of  in-depth  training  and  one-on-one 
assistance  from  advisors  and  industry  experts.  For  example,  as  part  of  its 
Commercialization  Pilot  Program,24  the  Army  assisted  selected  phase  II 
SBIR  award  recipients  in  assessing  commercialization  potential, 
developing  business  plans,  and  matching  their  technologies  with  potential 
government  and  industry  customers.  At  DOE,  staff  could  nominate  phase 
II  award  recipients  for  assistance  in  preparing  to  negotiate  business 
deals,  for  example,  including  joint  ventures  and  licensing  agreements  for 
use  of  their  technologies.  DOE  officials  told  us  that  in  2010  the  agency 
curtailed  its  use  of  SBIR  funds  for  phase  II  technical  assistance,  spending 
such  funds  on  assistance  only  for  award  recipients  that  had  specifically 
budgeted  for  it  in  their  applications.  In  2007  and  2008,  NASA  partnered 
with  the  Navy  to  pilot  a  technical  assistance  program  for  NASA  phase  II 
recipients.  The  program  was  designed  to  help  SBIR  businesses  develop 
a  plan  for  transitioning  to  phase  III,  among  other  things.  In  2007,  17 
phase  II  companies  with  19  SBIR  projects  participated  in  the  program, 
and  in  the  following  year,  19  phase  II  companies  with  20  SBIR  projects 
participated.  The  program  was  not  renewed  for  2009;  NASA  officials  told 
us  that  they  believed  the  program  was  generally  successful,  but  that  they 


24The  National  Defense  Authorization  Act  for  Fiscal  Year  2006,  Pub.  L.  No.  109-163,  § 
252,  119  Stat.  3136,  3178  (codified  as  amended  at  15  U.S.C.  §  638(y))  authorized  DOD’s 
Commercialization  Pilot  Program,  which  allows  DOD  components  to  use  a  portion  of  their 
SBIR  funds  for  program  administration  activities  that  support  accelerated  transition  of 
SBIR-funded  technologies  into  phase  III.  DOD  allows  its  components  discretion  to  use 
these  funds  for  their  own  specific  needs. 


Page  12 


GAO-1 1-698  Small  Business  Innovation  Research 


preferred  to  use  SBIR  funds  to  make  larger  awards.25  NIH  offered 
selected  current  or  past  phase  II  award  recipients  the  opportunity  to  work 
one-on-one  with  an  advisor  over  a  9-month  period  to  develop  business 
plans  to  commercialize  their  technologies,  as  well  as  to  prepare  materials 
to  help  attract  potential  investors  or  partners.  Since  2004,  almost  700 
award  recipients  have  received  the  assistance,  including  the  80  award 
recipients  currently  participating,  according  to  NIH  officials.  Finally,  NSF 
offered  customized  assistance  to  phase  II  award  recipients  through  its 
Innovation  Accelerator  Initiative.  According  to  NSF  officials,  through  this 
initiative,  award  recipients  received  help  in  connecting  with  potential 
investors  and  negotiating  company  acquisitions  and  mergers.  NSF 
officials  told  us  that,  in  2010,  approximately  33  percent  of  NSF’s  phase  II 
recipients  received  this  assistance. 

In  some  cases,  agencies  that  we  reviewed  used  non-SBIR  funds  to 
broaden  the  scope  of  the  technical  assistance  they  provided  to  help 
award  recipients  commercialize  their  technologies.26  For  example,  DOD 
used  non-SBIR  funds  to  host  its  annual  Beyond  Phase  II  Conference  and 
Technology  Showcase,  a  3-day  event  that  features  matchmaking 
sessions  with  SBIR  award  recipients  and  prime  contractors.  Similarly,  the 
Navy  used  non-SBIR  funds  to  maintain  databases  with  advanced 
searching  capability  to  help  award  recipients  identify  potential  business 
partners.  The  Navy  also  used  non-SBIR  funds  for  its  Transition 
Assistance  Program,  which  provides  individualized  help  with 
commercialization  planning,  culminating  in  a  conference  designed  to 
facilitate  interaction  with  potential  business  partners.  Moreover,  in  2011, 
the  National  Cancer  Institute  launched  its  Regulatory  Assistance  Program 
using  non-SBIR  funds.  According  to  agency  officials  and  information  from 
the  agency’s  Web  site,  this  program  provides  SBIR  award  recipients  time 
with  consultants  experienced  in  various  regulatory  requirements — such  as 
those  for  anticancer  therapies,  imaging  technologies,  and  medical 


25NASA  did  not  use  SBIR  funds  to  support  the  technical  assistance  pilot  program; 
however,  a  NASA  official  indicated  that,  if  the  agency  had  continued  the  program,  it  likely 
would  have  done  so  using  SBIR  funds.  NASA  officials  indicated  that  the  agency  has  relied 
on  other  resources  to  help  increase  commercialization,  such  as  national  SBIR 
conferences  hosted  by  states;  industry-sponsored  technology  summits;  the  Space 
Alliance  Technology  Outreach  Program,  which  is  designed  to  speed  the  transfer  of  space 
technology  to  the  private  sector;  non-SBIR-specific  agency  publications;  and  the  agency’s 
Web  site. 

26ln  this  report,  we  are  using  “non-SBIR  funds”  to  refer  to  funds  other  than  the  2.5  percent 
of  the  participating  agencies’  extramural  R&D  budgets  that  must  be  set  aside  for  SBIR. 
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devices — to  prepare  strategies  for  obtaining  regulatory  approvals  required 
before  the  technologies  can  be  commercialized.  The  National  Cancer 
Institute  also  used  non-SBIR  funds  to  support  its  Investor  Forum,  which 
provides  competitively  selected  SBIR  award  recipients  an  opportunity  to 
showcase  their  technologies  and  enter  into  discussions  with  the  biotech 
investment  community.  In  2010,  14  award  recipients  that  were  selected 
on  the  basis  of  strength  of  research,  impact  on  cancer,  product 
development,  and  market  potential  participated  in  the  forum  along  with 
more  than  175  potential  investors,  according  to  the  agency’s  Web  site. 

Matching  funds  programs.  Through  matching  funds  programs,  agencies 
provide  additional  SBIR  funds  to  award  recipients  that  obtain  monetary 
commitments  above  certain  thresholds  from  outside  investors.27  SBA  has 
designated  matching  funds  programs  as  a  best  practice  on  its  SBIR  Web 
site,  and  all  of  the  agencies  we  reviewed  except  DOE  have  established 
such  programs.28  For  example,  for  award  recipients  that  obtain  a 
minimum  of  $100,000  from  an  outside  investor,  NSF  will  match  up  to 
50  percent  of  the  outside  investment  for  a  maximum  of  $500,000  in  NSF 
matching  funds.  NASA  and  NIH  officials  said  that  matching  funds 
programs  encourage  outside  investment  during  the  early  stages  of 
R&D — a  time  when  many  investors  are  reluctant  to  invest.  In  particular, 
officials  at  the  National  Cancer  Institute  said  that  matching  funds  can  help 
attract  outside  investment  because  they  can  be  used  as  leverage  to 
increase  investors’  potential  returns.  DOD  and  NSF  offer  matching  funds 
to  award  recipients  at  the  end  of  phase  I  and  during  phase  II,  while  NASA 
and  the  National  Cancer  Institute  offer  matching  funds  during  phase  II. 
DOE  has  not  established  a  matching  funds  program  for  its  SBIR  program. 
DOE  officials  told  us,  however,  that  they  are  exploring  whether  to  do  so 
and  have  held  discussions  with  other  SBIR  participating  agencies  about 
their  matching  funds  programs. 

Officials  at  DOD,  NASA,  and  NIH  said  they  have  not  collected  data  to 
compare  the  commercialization  success  of  recipients  that  received 


270utside  investment  can  be  in  the  form  of  capital,  liquid  assets,  or  convertible  debt.  SBA 
has  identified  outside  (non-SBIR)  investment  as  a  metric  for  commercialization,  as 
discussed  later  in  this  report. 

oo 

NIH’s  National  Cancer  Institute  refers  to  its  matching  funds  awards  as  bridge  awards. 
The  National  Cancer  Institute  is  the  only  NIH  component  included  in  our  review  that 
offered  matching  funds. 
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matching  funds  with  the  success  of  those  that  did  not.29  NSF  conducted  a 
study  to  assess  the  effect  of  the  $18  million  in  matching  funds  it  invested 
in  fiscal  year  2006  for  48  phase  II  award  recipients  that  had  raised  a  total 
of  $58  million  from  outside  investors.30  According  to  NSF  officials,  results 
of  this  study  showed  that,  in  the  5  years  following  the  start  of  these  phase 
II  projects,  70  percent  of  recipients  that  had  received  matching  funds 
achieved  commercial  success  compared  with  a  30  percent  success  rate 
for  recipients  that  had  not  received  such  funds. 


Agencies  Generally 
Encouraged  Participation 
by  Disadvantaged 
Individuals  and  Women 
through  Outreach 
Activities  Directed  toward 
a  Broader  Audience 


SBA’s  guidance  states  that  small  businesses  owned  by  disadvantaged 
individuals  and  women  must  compete  for  SBIR  awards  on  the  same  basis 
as  all  other  small  businesses.  However,  to  meet  requirements  for  greater 
outreach  to  small  businesses  owned  by  disadvantaged  individuals  and 
women,  SBA  has  encouraged  participating  SBIR  agencies  to  reach  out  to 
such  businesses  and  to  develop  methods  that  encourage  their 
participation.  SBA  has  also  raised  the  topic  of  outreach  during  recent 
quarterly  meetings  of  agency  SBIR  program  managers.31  Officials  at  all  of 
the  agencies  we  reviewed  told  us  they  generally  reach  out  to  such 
businesses  through  activities  directed  toward  a  broader  audience,  such 
as  by  attending  SBIR  national  conferences  and  industry-sponsored 
events  and  by  sharing  information  via  Web  sites  or  e-mail  lists.  Agency 
officials  also  noted  that  they  try  to  accommodate  requests  for  speakers  at 
events  sponsored  by,  or  likely  to  be  attended  by,  small  businesses  owned 
by  disadvantaged  individuals  and  women — for  example,  events 
sponsored  by  trade  organizations  for  minority-  or  women-owned 
businesses.  However,  officials  from  some  trade  organizations  for 
businesses  owned  by  disadvantaged  individuals  and  women  told  us  that 


29NASA  and  the  National  Cancer  Institute  established  their  matching  funds  programs 
relatively  recently — in  2010  and  2009,  respectively.  DOD  began  matching  funds  for  phase 
I  award  recipients  in  1995  and  for  phase  II  award  recipients  in  2000. 

op 

NSF  established  a  matching  funds  program  for  its  phase  II  award  recipients  in  1998  and 
later  established  a  similar  program  for  phase  I  award  recipients  in  2007. 

31  In  addition,  in  2010  SBA  made  20  awards  to  state  organizations  though  the  Federal  and 
State  Technology  Partnership  Program.  Under  the  program,  SBA,  with  the  concurrence  of 
NSF  and  DOD  officials,  makes  awards  to  provide  outreach,  financial  support,  or  technical 
assistance  to  small  businesses  participating  in  or  interested  in  participating  in  the  SBIR 
program.  In  making  such  awards,  the  agencies  are  required  to  consider  whether  the 
applications  being  funded  address  the  needs  of  small  businesses  owned  by  minorities  or 
women,  among  other  things. 
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the  outreach  of  agencies  we  reviewed  was  often  ineffective  in  educating 
the  organizations’  members  about  the  SBIR  program. 

Of  the  agencies  we  reviewed,  NIH  and  NSF  have  made  specific  efforts, 
including  the  following,  to  improve  their  outreach: 

•  For  fiscal  years  2010  and  2011,  NIH  developed  a  goal  to  increase 
awareness  of  its  SBIR  program  among  businesses  owned  by 
disadvantaged  individuals  and  women,  and  it  outlined  specific 
activities  aimed  at  reaching  this  goal. 

•  Both  NIH  and  NSF  offered  various  fellowships  for  postdoctoral 
research  conducted  by  disadvantaged  individuals  and  women;  these 
fellowships  were  available  to  support  SBIR  projects,  as  well  as  other 
research. 

•  In  2010,  NSF  assigned  a  full-time  staff  member  to  help  it  develop  a 
plan  to  increase  participation  in  SBIR  by  businesses  owned  by 
disadvantaged  individuals  and  women  in  response  to  a 
recommendation  from  its  SBIR  advisory  committee. 

•  Through  a  review  of  academic  literature,  as  well  as  informal  polling  of 
NSF  applicants  and  award  recipients,  NSF  has  identified  barriers  to 
SBIR  participation  by  small  businesses  owned  by  disadvantaged 
individuals  and  women,  NSF  officials  told  us.  These  barriers  include 
disparities  in  the  owners’  levels  of  education  and  access  to  capital 
compared  with  those  of  other  entrepreneurs.  To  address  identified 
barriers,  NSF  is,  among  other  things,  establishing  partnerships  with 
industry  and  academia  to  expose  African  American,  Latino,  and  other 
college  students  to  entrepreneurship  in  scientific  and  technical  fields, 
according  to  NSF  officials. 

Evaluation  of  the  effectiveness  of  agencies’  outreach  efforts  is  hindered 
by  a  lack  of  accurate  and  complete  data.  Although  SBA  collects  data  on 
the  number  and  dollar  value  of  awards  to  small  businesses  owned  by 
disadvantaged  individuals  and  women,  SBA  officials  told  us  that  they 
cannot  accurately  tabulate  data  on  such  awards,  particularly  awards  to 
women-owned  businesses,  because  of  inconsistencies  in  the  data  on 
business  ownership.  According  to  the  officials,  SBA  has  taken  steps  to 
correct  the  inconsistencies  for  data  submitted  after  2006  but  has  not  done 
so  for  earlier  years.  Moreover,  SBA  does  not  collect  data  on  the  number 
of  applications  submitted  by  businesses  owned  by  disadvantaged 
individuals  and  women.  As  a  result,  SBA’s  data  do  not  allow  for  an 
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examination  of  trends  in  the  submission  of  applications  from  such 
businesses,  analysis  of  the  percentage  of  applications  from  these 
businesses  that  lead  to  awards,  or  correlation  of  these  trends  and 
percentages  with  outreach  efforts.  SBA  officials  told  us  in  March  201 1 
that  they  were  considering  whether  their  database  should  include 
information  on  the  numbers  of  applications  submitted  by  these 
businesses. 


Comparable  Data  Are 
Not  Available  across 
Participating  Agencies 
to  Evaluate  Progress 
in  Increasing 
Commercialization  of 
SBIR  Technologies 


SBA  has  not  yet  developed  the  government-use  portion  of  its  database 
for  collecting  comparable  commercialization  data  on  SBIR  technologies, 
but  it  is  taking  steps  to  do  so.  In  the  interim,  agencies  have,  for  their  own 
purposes,  independently  gathered  commercialization  data  that  are  not 
comparable;  the  accuracy  of  these  data  is  largely  unknown.  Implementing 
the  government-use  portion  of  the  database  should  improve  the 
comparability  of  the  data.  However,  programwide  evaluation  of  progress 
in  increasing  commercialization  may  continue  to  be  impaired  by  long¬ 
standing  challenges. 


SBA  Has  Not  Implemented 
the  Government-Use 
Portion  of  Its  Database 
Intended  to  Collect 
Comparable 

Commercialization  Data 
for  Program  Evaluation 
but  Is  Taking  Steps  to 
Do  So 


As  of  June  201 1 ,  SBA  had  not  met  the  legislative  mandate  to  develop  and 
implement,  by  June  2001,  a  government-use  database  that  can  provide 
data  on  commercialization  for  evaluating  the  SBIR  program.32  However, 
the  agency’s  efforts  to  develop  such  a  database  recently  gained 
additional  prominence  and  resources.  Specifically,  SBA  linked 
development  of  the  government-use  portion  of  its  database  to  one  of  the 
agency’s  high-priority  performance  goals  for  fiscal  years  201 1  and  2012. 
Additionally,  in  September  2010,  SBA  allocated  $1.4  million  in  Recovery 
Act  funds  to  hire  a  new  contractor  to  develop  the  government-use 
portion’s  capacity  to  accept  commercialization  data  submitted  by 
participating  SBIR  agencies  and  award  recipients,  as  well  as  to  make 
other  improvements  to  the  database.  For  example,  SBA  said  that  it  has 
been  working  with  the  contractor  to  consolidate  data  on  previous  awards. 
SBA  officials  said  that  past  award  recipients  have  been  assigned  unique 
identifiers  that  will  be  used  to  track  awards  issued  to  those  recipients  over 
the  lifetime  of  the  SBIR  program;  unique  identifiers  are  also  to  be 
assigned  to  small  businesses  newly  entering  the  program.  In  the  future, 


3215  U.S.C.  §  638(k)(2)  (2006). 
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SBA  intends  for  the  unique  identifiers  to  allow  agencies  to  validate 
business  information  by  comparing  it  against  information  in  other  federal 
databases  such  as  the  Central  Contractor  Registration  database,  which 
contains  information  on  businesses  that  want  to  contract  with  the  federal 
government.33  SBA  officials  told  us  that  they  expect  to  implement  the 
government-use  portion  of  SBA’s  database  by  August  2011  and  to 
provide  for  its  basic  maintenance  and  support  despite  reductions  in  the 
agency’s  overall  budget.34 

The  government-use  portion  is  intended  to  allow  both  participating 
agencies  and  award  recipients  to  enter  commercialization  data  in  a 
comparable  format  to  assist  in  program  evaluation.  SBA  officials  told  us 
that  they  have  worked  with  participating  agencies  to  develop  common 
metrics  for  commercialization  data,  as  well  as  a  standardized  data 
collection  instrument  that  will  accommodate  the  various  types  of  SBIR 
technologies  the  agencies  fund  to  meet  their  different  missions.  These 
metrics,  which  will  correspond  to  fields  in  the  database,  include  the 
following: 

•  indication  of  whether  an  award  resulted  in  a  commercialized 
technology  and  whether  other  SBIR  awards  contributed  to 
commercialization  of  the  technology; 

•  estimated  sales; 

•  estimated  investment  (other  than  SBIR  funding); 

•  any  patents  applied  for  or  received  related  to  the  award;  and 

•  any  initial  public  offering,  merger,  or  sale  of  the  business  that  resulted, 
at  least  in  part,  from  the  award. 


oo 

Businesses  are  required  to  provide  information  for  the  Central  Contractor  Registration 
database  such  as  their  Employer  Identification  Number,  legal  business  name,  and  the 
goods  and  services  they  provide. 

34SBA  officials  noted  that  budgetary  uncertainties  may  affect  related  efforts.  For  example, 
the  officials  said  that  SBA  has  delayed  plans  to  develop  a  commercialization  survey  for 
award  recipients  that  do  not  apply  for  a  new  SBIR  award  within  1  year  after  their  phase  II 
award  ends. 
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SBA  officials  told  us  in  May  201 1  that  they  plan  to  implement  the  metrics 
and  data  collection  instrument  in  August  201 1 .35 

SBA  is  requesting  that  participating  agencies  voluntarily  begin  entering 
historical  commercialization  data  into  the  government-use  database 
before  August  201 1 .  To  facilitate  this  process,  SBA  is  working  with  its 
contractor  to  ensure  that  historical  agency  data  can  be  matched  to  fields 
in  the  new  database.  Nevertheless,  officials  from  SBA  and  participating 
SBIR  agencies  said  that  some  agencies  may  not  enter  historical  data  or 
may  be  delayed  in  doing  so  because  they  either  did  not  collect  such  data 
or  do  not  have  the  data  in  electronic  form.  For  example,  NASA  officials 
stated  that  much  of  their  commercialization  data  are  stored  in  paper 
format  and  expressed  doubt  that  the  agency  would  be  able  to  convert  the 
data  into  the  required  format  for  entering  by  SBA’s  August  deadline. 

SBA  officials  also  told  us  that,  after  the  government-use  portion  of  the 
database  is  available,  some  agencies  may  instruct  applicants  and  award 
recipients  to  submit  their  commercialization  data  directly  into  the 
database.  Other  agencies,  such  as  DOD,  may  continue  to  require 
applicants  and  recipients  to  submit  commercialization  data  directly  to  the 
agencies,  which  would  then  upload  the  data  into  the  database.  As  of  May 
2011,  SBA  officials  were  unsure  which  approach  agencies  would  take, 
noting  that  agencies  may  wait  to  see  how  the  database  works  before 
making  a  decision. 


Without  the  Government- 
Use  Portion  of  the  SBA 
Database,  Agencies  Have 
Independently  Collected 
Commercialization  Data 
That  Are  Not  Comparable 


In  the  absence  of  the  government-use  portion  of  SBA’s  database,  the  five 
participating  SBIR  agencies  we  reviewed  have  independently  collected 
commercialization  data  that  are  not  comparable.  The  agencies  collected 
these  data  using  various  methods  for  their  own  purposes,  as  summarized 
in  table  1 . 


35SBA  officials  said  that  the  agency  had  previously  obtained  approval  from  the  Office  of 
Management  and  Budget,  as  required  by  the  Paperwork  Reduction  Act,  for  data  collection 
instruments  that  are  substantially  similar  to  the  ones  proposed,  and  they  plan  to  rely  on 
this  approval.  Paperwork  Reduction  Act,  44  U.S.C.  §§  3501-3520  (2006). 
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Table  1: 

Summary  of  Five  SBIR  Agencies’  Efforts  to  Gather  Commercialization  Data 

Agency 

Data  collection  instrument 

Dates  administered 

Population  queried 

Purpose  of  data  collection 

DOD 

Commercialization  report 
submitted  as  a  component  of 
applications  for  SBIR  awards3 

2000  to  present 

Applicants  for  phase  1  and 
phase  II  awards 

To  assess  an  applicant’s 
ability  to  commercialize  SBIR 
technologies,  which  DOD 
considers  when  selecting 
projects  for  funding,  and  to 
assess  the  extent  to  which 
DOD’s  SBIR-funded 
technologies  are 
commercialized 

DOE 

Annual  survey 

1986-2007b 

Award  recipients  with  active 
phase  II  awards3 

To  assess  the 

commercialization  success  of 
DOE’s  SBIR-funded  projects 

NASA 

Annual  survey 

1997-2002d 

Award  recipients  that 
received  a  phase  II  award 
from  1983  through  1996 

T o  assess  the  effect  of 
changes  that  NASA  made  in 
1995  to  its  SBIR  program 
management  and  application 
review  process — i.e.,  the  effect 
of  these  changes  on  the 
success  of  award  recipients  in 
commercializing  technologies 
and  in  integrating  those 
technologies  into  the  agency’s 
mission  programs 

NIH 

Periodic  survey 

2002  (with  follow-up 
surveys  of  2002 
respondents  in  2004, 
2005,  and  2007);  2008 

2002  and  follow-up  surveys: 
award  recipients  that 
received  phase  II  awards 
from  1992  through  2001 

2008:  award  recipients  that 
received  phase  II  awards 
from  2002  through  2006 

To  evaluate  the  extent  to 
which  NIH’s  program  met  the 
overall  program  goals, 
including  the  extent  to  which  it 
increased  commercialization 
of  SBIR  technologies  that  the 
agency  funded 

NSF 

Periodic  survey 

2005-present 

Phase  II  award  recipients 
marking  the  3rd,  5th,  and 

8th  anniversary  of  the 
receipt  of  their  awards 

To  help  assess  the 
commercialization  success  of 
NSF’s  SBIR  program 

Source:  GAO  analysis  of  agency  data. 


aSmall  businesses  applying  for  DOD  SBIR  awards  must  report  on  the  commercialization  history  of 
any  prior  phase  II  SBIR  awards  received. 

bAccording  to  DOE  officials,  they  discontinued  the  annual  survey  for  a  combination  of  reasons, 
including  contractor  staff  changes  and  resource  constraints.  However,  DOE  has  indicated  that  it 
expects  to  resume  surveying  in  fall  201 1  using  an  improved  survey  instrument. 

cDOE  award  recipients  were  surveyed  during  the  years  in  which  their  phase  II  awards  were  active 
and  for  up  to  5  years  after  the  awards  ended. 

dAccording  to  NASA  officials,  the  agency  has  not  conducted  a  follow-up  survey  owing  to  a  lack  of 
resources,  although  the  agency  has  continued  to  link  its  survey  instrument  to  its  SBIR  solicitations  for 
applicants  to  complete  voluntarily.  However,  agency  officials  noted  that  response  rates  have  been  low. 
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In  conducting  their  data  collection  efforts,  agencies  differed  in  the  extent 
to  which  they  asked  award  recipients  to  do  the  following,  among  other 
things: 


•  Identify  the  type  of  customer  and  the  amount  of  sales  or  further 
investment  for  SBIR-funded  technologies.  For  example,  most 
agencies  asked  award  recipients  to  report  federal  and  nonfederal 
sales  separately,  but  NIH  and  NSF  asked  award  recipients  to  report 
combined  sales. 

•  Account  for  indirect  sales  and  nonfinancial  indicators  of 
commercialization.  NASA,  NIH,  and  NSF  asked  award  recipients  to 
indicate  whether  an  SBIR-funded  technology  had  resulted  in  licensing 
agreements  with  other  businesses  to  sell  the  technology,  while  DOD 
and  DOE  did  not  ask  that  question.  NASA  further  asked  award 
recipients  to  estimate  the  financial  value  of  such  agreements,  while 
the  other  agencies  did  not.  Similarly  NASA,  NIH,  and  NSF  asked 
award  recipients  to  indicate  whether  specific  SBIR-funded 
technologies  had  resulted  in  patents,  while  DOD  and  DOE  asked 
award  recipients  to  report  the  total  number  of  patents  resulting  from  all 
their  SBIR  awards. 

•  Quantify  the  dollar  values  of  cumulative  sales.  While  most  agencies 
asked  award  recipients  to  report  a  specific  dollar  amount  in 
cumulative  sales  resulting  from  their  SBIR-funded  technologies  over  a 
period  of  time,  NIH  asked  award  recipients  to  report  such  sales  by 
choosing  among  ranges,  beginning  with  “$50,000  or  less”  and 
extending  to  “$50,000,000  or  more.”  Because  NIH  has  reported 
cumulative  sales  in  ranges  rather  than  specific  dollar  amounts, 
comparing  its  results  with  those  reported  by  other  agencies  is  difficult. 

While  each  agency’s  data  collection  efforts  resulted  in,  among  other 
information,  estimates  of  total  or  average  sales  of  SBIR  technologies, 
differences  in  the  agencies’  data  collection  efforts  make  it  difficult  to 
compare  results  across  agencies.  The  following  are  examples  of 
commercialization  data  reported  by  agencies: 

•  DOD  estimated  that  commercialization  of  SBIR  technologies  that  it 
funded  generated  federal  and  nonfederal  sales  and  non-SBIR  funding 
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of  $22  billion  on  a  program  investment  of  $1 1  billion  from  2000 
through  March  2010. 36 

•  DOE  estimated  that,  from  1986  through  2007,  SBIR  technologies 
developed  by  recipients  of  phase  II  awards  resulted  in  a  total  of  $2.4 
billion  in  federal  and  nonfederal  sales  and  $1.6  billion  in  non-SBIR 
investment.  On  average,  award  recipients  reported  receiving  more 
than  $3  million  in  sales  related  to  SBIR-funded  technologies.  During 
the  same  period,  DOE  reported  that  it  had  invested  $1.6  billion  in 
phase  I  and  II  SBIR  awards. 

•  NASA  estimated  that,  as  of  2002,  SBIR  technologies  developed  by 
award  recipients  that  received  a  phase  II  award  from  1983  through 
1996  had  generated  approximately  $2.8  billion  in  federal  and 
nonfederal  sales  and  non-SBIR  funding  compared  with  $1.1  billion  in 
SBIR  investment  from  NASA. 

•  In  NIH’s  2002  survey,  which  covered  1992  through  2001, 27  percent 
of  respondents  reported  an  estimated  total  of  $821  million  in  sales  of 
SBIR  technologies;  the  other  respondents  did  not  report  any  sales.37 
NIH  estimated  that  it  invested  $2.2  billion  in  phase  I  and  phase  II 
awards  from  1992  through  2001.  For  the  2008  survey,  which  covered 
2002  through  2006,  33  percent  of  respondents  reported  an  estimated 
total  of  $396  million  in  federal  and  nonfederal  sales  of  SBIR 


36DOD’s  estimate  of  returns  on  investment  also  includes  the  commercialization  results  of 
awards  made  through  DOD’s  Small  Business  Technology  Transfer  (STTR)  program.  The 
STTR  program  resembles  the  SBIR  program  in  some  respects  but  provides  funding  for 
research  proposals  that  are  developed  and  executed  cooperatively  between  small 
businesses  and  research  organizations. 

37For  NIH’s  2002  survey,  768  (73  percent)  of  the  1,052  recipients  of  phase  II  awards  from 
1992  through  2001  were  located  and  responded  to  the  survey.  Follow-up  surveys 
experienced  declining  responses  from  recipients.  Of  the  768  respondents  to  the  2002 
survey,  275  (36  percent)  were  located  and  responded  to  the  final  follow-up  survey  in  2007 
In  reporting  survey  results,  NIH  calculated  its  response  rate  from  the  sample  of  award 
recipients  that  it  deemed  usable  and  eligible — that  is,  those  award  recipients  that,  among 
other  criteria,  could  be  located  and  were  still  operating  in  the  United  States.  For  the  2002 
survey,  NIH  used  the  size  of  this  sample — 905  contacts  out  of  the  1 ,052  award 
recipients — to  determine  that  the  768  respondents  represented  a  survey  response  rate  of 
85  percent. 
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technologies.38  NIH  estimated  that  it  invested  $2.7  billion  in  phase  I 
and  phase  II  awards  from  2002  through  2006.  NIH  was  the  only 
agency  we  reviewed  that  reported  sales  lower  than  its  SBIR 
investment  for  the  periods  it  examined.  According  to  NIH  officials, 
many  of  the  technologies  that  the  agency  supports  through  its  SBIR 
program,  such  as  drugs  and  medical  devices,  take  longer  to 
commercialize  than  those  funded  by  other  agencies  because  of  the 
need  for  extensive  clinical  testing  and  regulatory  approval. 

•  NSF  officials  estimated  that  recipients  marking  the  eighth  anniversary 
of  the  receipt  of  their  awards  from  July  2005  through  May  2010  had 
realized  a  total  of  $1.05  billion  in  commercial  revenue.39  NSF 
estimated  that  it  invested  $628  million  in  SBIR  awards  during  roughly 
the  same  period.40 

Further,  with  the  exception  of  DOD,  agencies  we  reviewed  generally  did 
not  take  steps  to  verify  commercialization  data  that  they  received  from 
award  recipients,  so  the  accuracy  of  the  data  is  largely  unknown.  As 
officials  from  some  of  the  agencies  in  our  review  noted,  award  recipients 
may  have  an  incentive  to  overstate  their  commercialization  success  in  the 
hope  of  improving  their  prospects  of  receiving  future  SBIR  awards.  While 
SBA  has  worked  with  SBIR  agencies  to  identify  best  practices  in  other 
areas  of  SBIR  program  management,  it  has  not  identified  best  practices 
for  agencies  to  use  in  verifying  the  accuracy  of  commercialization  data. 
Without  consistent  practices  for  verifying  the  accuracy  of  these  data,  the 
usefulness  of  the  government-use  portion  of  SBA’s  database  as  a  tool  for 
evaluating  the  SBIR  program’s  success  in  increasing  commercialization 
may  be  limited. 


oo 

For  NIH’s  2008  survey,  719  (69  percent)  of  the  1,037  recipients  of  phase  II  awards  from 
2002  through  2006  were  located  and  responded  to  the  survey.  For  the  2008  survey,  NIH 
used  the  size  of  the  sample  deemed  usable  and  eligible — 918  out  of  the  1 ,037  award 
recipients — to  determine  that  the  719  respondents  represented  a  response  rate  of  78 
percent. 

on 

NSF’s  estimate  of  commercial  revenue  includes  sales  revenue  from  SBIR-funded 
technologies;  licensing  revenue;  and  revenue  from  mergers  or  sales  of  the  award 
recipients’  businesses  that  resulted,  at  least  in  part,  from  the  awards.  NSF’s  estimate  does 
not  include  venture  capital  funding. 

40NSF’s  estimated  investment  covers  the  period  from  October  2004  through  October  201 0. 
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To  verify  the  accuracy  of  award  recipients’  commercialization  data,  DOD 
performs  an  annual  review  of  all  projects  in  its  Company 
Commercialization  Database,  which  contains  the  commercialization  data 
it  gathers  from  award  recipients.  This  review  includes  checks  to  ensure 
that  prior  award  recipients  applying  for  new  awards  are  not  reporting  the 
same  project  results  more  than  once,  substituting  the  results  of  one 
project  for  that  of  another,  or  incorrectly  reporting  sales  to  third  parties. 
According  to  DOD  officials,  after  its  2010  review,  the  agency  sent 
approximately  300  e-mail  queries  to  applicants  whose  reported 
commercialization  data  were  identified  as  having  potential  problems.  The 
officials  said  that  applicants  that  do  not  respond  to  such  queries  are 
blocked  from  submitting  further  applications  until  concerns  related  to  their 
commercialization  reports  are  addressed.  Even  with  these  verification 
activities,  however,  Army  officials  expressed  concern  to  us  about  the 
accuracy  of  the  applicants’  self-reported  commercialization  data;  these 
officials  stated  their  preference  for  using  data  from  the  Federal 
Procurement  Data  System,  which  contains  government  information  on 
federal  contracts,  including  sales.41  Moreover,  a  Navy  official 
acknowledged  the  possibility  that  additional  verification  activities,  such  as 
selective  spot  visits  to  SBIR  award  recipients,  could  further  deter 
recipients  from  misrepresenting  their  commercialization  success, 
although  he  noted  that  such  activities  would  compete  with  other 
administrative  priorities.  Similarly,  officials  from  DOE  and  NIH  stated  that 
additional  verification  activities  would  be  useful  but  also  said  that  they 
needed  to  devote  program  administration  resources  to  higher  priority 
activities,  such  as  preparing  solicitations  and  supporting  review  panels  for 
applications.42 


41  As  we  reported  in  2010,  commercialization  data  are  not  always  correctly  entered  into 
this  database  by  DOD  procurement  officers.  See  GAO,  Space  Acquisitions:  Challenges  in 
Commercializing  Technologies  Developed  under  the  Small  Business  Innovation  Research 
Program,  GAO-11-21  (Washington,  D.C.:  Nov.  10,  2010).  However,  DOD  officials  said 
that  they  are  improving  data  in  this  system  by  training  officers  to  correctly  account  for 
SBIR  contracts. 

42Agencies  are  prohibited  from  using  SBIR  funds  for  program  administration  with  the 
exception  of  certain  administrative  activities  authorized  as  part  of  DOD’s 
Commercialization  Pilot  Program. 
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Implementing  the 
Government-Use  Portion 
of  the  Database  Should 
Improve  the  Comparability 
of  Commercialization 
Data,  but  Long-standing 
Challenges  May  Still 
Impair  Evaluation  of 
Commercialization 
Progress 


SBA’s  implementation  of  the  government-use  portion  of  its  database 
should  improve  the  comparability  of  commercialization  data  available  for 
programwide  evaluation.  Nevertheless,  long-standing  challenges  may 
continue  to  impair  programwide  evaluation  of  progress  in  increasing 
commercialization  of  SBIR-funded  technologies.  As  we  reported  in 
October  2006,  notable  among  these  challenges  is  that  prior  award 
recipients  that  are  no  longer  participating  in  the  SBIR  program  are  not 
required  to  provide  updated  commercialization  data  and  may  prefer  not  to 
do  so.43  For  example,  DOD  indicated  in  written  comments  to  us  that,  from 
2008  to  2010,  46  percent  of  nonparticipating  prior  phase  II  award 
recipients  did  not  provide  updates  despite  DOD’s  request  that  they 
update  commercialization  data  annually  after  their  awards  ended.44 
Similarly,  in  a  report  on  its  2002  survey,  NASA  observed  that  many 
recipients  of  multiple  awards  elected  not  to  respond  to  its  survey  despite 
“extensive  telephone  follow-up”  and  that  many  recipients  that  ultimately 
responded  “would  likely  have  preferred  not  to.”45  Some  award  recipients 
may  be  reluctant  to  provide  commercialization  data  because  the  data  are 
business-sensitive.  SBA  officials  told  us  that  mechanisms  to  require  or 
encourage  nonparticipating  recipients  to  report  their  data  need  to  be 
explored.  A  NASA  official  told  us  that  effective  incentives  to  encourage 
wider  voluntary  reporting  might  include  publicizing  commercial  success  or 
giving  monetary  prizes  for  success. 


The  difficulties  agencies  face  in  persuading  prior  award  recipients  to 
volunteer  commercialization  information  can  be  compounded  by 
challenges  in  maintaining  contact  with  them.  Specifically,  prior  award 
recipients  can  change  names  or  personnel,  go  out  of  business,  or  be  sold 
during  the  10  or  more  years  that  it  can  take  for  an  SBIR-funded 
technology  to  reach  the  marketplace.  In  NIH’s  2002  survey  of  award 
recipients,  for  example,  the  portion  of  the  sample  that  was  “unusable” — a 
group  that  consisted  primarily  of  recipients  that  no  longer  existed  or  could 
not  be  found — increased  from  2  percent  in  the  first  year  after  the  end  of 
the  award  to  52  percent  in  the  tenth  year. 


43GAO-07-38. 

44According  to  DOD,  this  percentage  does  not  necessarily  represent  historic  averages. 

45NASA’s  survey,  conducted  from  1997  through  2002,  achieved  a  78  percent  response 
rate. 


Page  25 


GAO-1 1-698  Small  Business  Innovation  Research 


Programwide  evaluation — particularly  efforts  to  compare 
commercialization  success  across  agencies — can  also  be  complicated  by 
differences  in  the  time  required  to  commercialize  various  types  of  SBIR- 
funded  technologies.  Comparing  agencies’  commercialization  results  at  a 
given  point  in  time  may  not  present  a  true  picture  of  each  agency’s 
success  because  some  agencies  fund  technologies  that  are  relatively 
close  to  being  market-ready  while  others  fund  technologies  that  need 
more  extensive  development  or  regulatory  approval.  Furthermore,  as  we 
have  previously  reported,  the  SBIR  program’s  other  goals  remain 
important,  and  comparisons  that  focus  on  commercialization  may  not 
adequately  take  into  account  progress  toward  these  goals.46  For 
example,  one  agency  official  told  us  that  some  SBIR-funded  technologies, 
such  as  those  related  to  national  security,  may  never  have  great 
commercial  potential  but  are  important  to  the  agency’s  mission. 

Conclusions 

DOD,  DOE,  NASA,  NIH,  and  NSF  have  designed  their  SBIR  solicitations 
to  address  the  program’s  purposes  of  using  small  businesses  to  meet 
federal  R&D  needs,  stimulating  technological  innovation,  and  increasing 
commercialization  of  innovations  derived  from  federal  R&D  efforts,  and 
they  have  further  addressed  commercialization  by  providing  technical 
assistance  or  matching  funds  to  award  recipients.  These  agencies  have 
also  conducted  outreach  and  other  activities  to  address  the  SBIR  purpose 
of  encouraging  participation  in  technological  innovation  by  small 
businesses  owned  by  disadvantaged  individuals  and  women.  However, 
evaluation  of  progress  in  achieving  the  program’s  purposes  is  impeded  by 
a  lack  of  accurate,  comparable,  and  complete  data  on  program  results. 

For  example,  it  is  difficult  to  evaluate  the  program’s  effectiveness  in 
encouraging  small  businesses  owned  by  disadvantaged  individuals  or 
women  to  participate  in  technological  innovation  because  SBA  does  not 
collect  data  on  the  number  of  applications  submitted  by  such  businesses. 

It  is  also  difficult  to  evaluate  the  program’s  effectiveness  in  increasing 
commercialization  of  SBIR-funded  technologies  because,  although 
agencies  participating  in  the  program  have  gathered  commercialization 
data  for  their  own  purposes,  comparable  data  on  commercialization  are 
not  available  across  agencies.  SBA’s  planned  implementation  of  a 
government-use  portion  of  its  database  should  go  some  way  toward 
improving  the  comparability  of  the  commercialization  data  as  they  are 

46GAO/RCED-99-1 14. 
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systematically  collected  using  common  metrics.  However,  the 
commercialization  data  that  the  database  is  intended  to  contain  are 
largely  self-reported  by  award  recipients  that  may  have  an  incentive  to 
overstate  their  commercialization  success.  DOD  has  adopted  practices 
for  verifying  the  accuracy  of  commercialization  data  it  collects  from  prior 
award  recipients,  but  most  of  the  participating  agencies  we  reviewed  did 
not  verify  the  accuracy  of  commercialization  data  from  their  prior  award 
recipients,  and  SBA  has  not  identified  best  practices  for  participating 
agencies  to  use  in  doing  so.  As  long  as  participating  agencies  do  not 
consistently  verify  the  accuracy  of  commercialization  data,  the  usefulness 
of  the  government-use  portion  of  SBA’s  database  as  a  tool  for  evaluating 
the  SBIR  program’s  success  in  increasing  commercialization  may  be 
limited. 


Recommendations  for 
Executive  Action 

•  collect  data  on  the  number  of  applications  submitted  by  small 
businesses  owned  by  disadvantaged  individuals  and  women,  and 

•  identify  best  practices  for  verifying  the  accuracy  of  data  related  to 
progress  in  increasing  commercialization. 


To  build  upon  efforts  to  implement  a  government-use  database  for 
program  evaluation,  we  recommend  that  the  Administrator  of  the  Small 
Business  Administration  work  with  participating  SBIR  agencies  to  take  the 
following  two  actions: 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  draft  of  this  report  to  SBA,  the  Departments  of  Defense 
and  Energy,  the  National  Aeronautics  and  Space  Administration,  the 
National  Institutes  of  Health,  and  the  National  Science  Foundation  for 
review  and  comment.  SBA  generally  agreed  with  our  findings  as  well  as 
our  recommendations,  which  it  offered  an  action  plan  to  address. 
Specifically,  with  respect  to  our  first  recommendation,  SBA  stated  that, 
beginning  in  fiscal  year  2012,  it  plans  to  use  its  database  to  collect 
information  from  agencies  about  applicants  that  did  not  receive  awards — 
information  that  could  include  whether  the  applicants  were  small 
businesses  owned  by  disadvantaged  individuals  or  women.  Further,  SBA 
indicated  that  it  plans  to  hold  a  workshop  in  fall  201 1  for  participating 
SBIR  agencies  to  share  best  practices  for  reaching  out  to  small 
businesses  owned  by  disadvantaged  individuals.  According  to  SBA,  the 
workshop  should  result  in  a  commitment  from  agencies  to  develop 
baselines  for  numbers  of  applications  from  such  businesses.  Regarding 
our  second  recommendation,  SBA  indicated  that  it  will  seek  to  identify 
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best  practices  and  methods  for  verifying  the  accuracy  of 
commercialization  data  and  will  work  with  agencies  toward 
implementation  of  those  practices  and  methods.  SBA  also  noted  that  its 
effort  to  collect  commercialization  data  is  intended  to  establish  a  baseline, 
against  which  SBA  can  review  progress  in  increasing  commercialization. 
SBA’s  letter  conveying  its  comments  is  contained  in  appendix  II. 

Among  the  SBIR  participating  agencies  that  we  reviewed,  DOE  and  NSF 
concurred  with  our  recommendations  and  provided  general  comments, 
which  are  included  in  appendixes  III  and  IV,  respectively.  Both  DOE  and 
NSF  also  made  technical  comments,  which  we  have  incorporated  into  our 
report  as  appropriate.  In  its  general  comments,  DOE  stated  that  it  collects 
information  on  the  number  of  applications  submitted  by  small  businesses 
owned  by  disadvantaged  individuals  and  women  and  is  willing  to  report 
the  data  to  SBA.  DOE  further  stated  that  it  does  not  verify 
commercialization  data  because  of  resource  limitations — not  a  belief  that 
verification  is  of  limited  value — and  it  expressed  an  interest  in  learning 
about  best  practices  for  verification  of  these  data.  In  addition,  DOE 
commented  that,  until  universal  metrics  are  identified  for  measuring  the 
success  of  SBIR  programs  across  agencies,  the  compatibility  of  available 
data  among  agencies  will  remain  a  secondary  concern.  NSF  stated  that  it 
concurs  with  the  underlying  goals  of  our  recommendations.  Moreover, 
NSF  affirmed  its  commitment  to  implementation  of  a  government-use 
database  for  program  evaluation,  collection  of  data  on  participation  in 
small  business  innovation,  and  identification  of  best  practices  for 
verification  of  commercialization  data.  The  remaining  agencies — DOD, 
HHS,  and  NASA — neither  agreed  nor  disagreed  with  our 
recommendations  but  provided  technical  comments,  which  we  have 
incorporated  into  our  report  as  appropriate. 
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As  agreed  with  your  offices,  unless  you  publicly  announce  the  contents  of 
this  report  earlier,  we  plan  no  further  distribution  until  30  days  from  the 
report  date.  At  that  time,  we  will  send  copies  to  the  appropriate 
congressional  committees,  Administrators  of  SBA  and  NASA,  Secretaries 
of  Defense  and  Energy,  Directors  of  NIH  and  NSF,  and  other  interested 
parties.  In  addition,  this  report  will  be  available  at  no  charge  on  the  GAO 
Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  members  have  any  questions  about  this  report,  please 
contact  me  at  (202)  512-3841  or  ruscof@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  Key  contributors  to  this  report  are  listed  in 
appendix  V. 


Frank  Rusco 

Director,  Natural  Resources 
and  Environment 
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Appendix  I:  Scope  and  Methodology 


In  conducting  this  study,  we  reviewed  Small  Business  Innovation 
Research  (SBIR)  program-related  activities  of  the  Small  Business 
Administration  (SBA)  and  5  of  the  11  SBIR  participating  agencies — the 
Department  of  Defense  (DOD),  Department  of  Energy,  National 
Aeronautics  and  Space  Administration,  Department  of  Health  and  Human 
Services’  National  Institutes  of  Health  (NIH), 1  and  National  Science 
Foundation  (NSF).  For  the  two  agencies  with  the  largest  SBIR  budgets — 
DOD  and  NIH — we  reviewed  program  activities  conducted  by  the  three 
participating  subcomponent  agencies  with  the  largest  SBIR  budgets 
because  some  key  activities  are  carried  out  at  that  level.  Specifically,  for 
DOD,  we  examined  the  SBIR  programs  of  the  Army,  Air  Force,  and  Navy, 
and  for  NIH,  we  examined  the  programs  of  the  National  Institute  of  Allergy 
and  Infectious  Diseases;  the  National  Cancer  Institute;  and  the  National 
Heart,  Lung,  and  Blood  Institute.  The  five  participating  agencies  we 
reviewed  accounted  for  about  96  percent  of  the  total  dollars  awarded  by 
the  program  in  fiscal  year  2009.  We  reviewed  applicable  laws  and 
regulations  and  literature  on  the  SBIR  program,  including  our  prior  reports 
and  assessments  by  a  committee  of  the  National  Academy  of  Sciences’ 
National  Research  Council.  To  obtain  further  context  for  our  review,  we 
attended  two  national  conferences  and  a  National  Research  Council 
workshop  on  the  SBIR  program,  and  we  interviewed  National  Research 
Council  staff  with  program  expertise. 

More  specifically,  to  determine  how  participating  agencies  have 
addressed  the  SBIR  program’s  four  overarching  purposes  when 
implementing  their  programs,  we  reviewed  SBA  documents  and  data, 
including  SBA’s  policy  directive  on  implementation  of  the  SBIR  program, 
minutes  from  selected  meetings  of  SBA  and  SBIR  program  directors, 
SBA’s  SBIR  annual  report  for  fiscal  year  2008  (the  latest  year  for  which 
an  annual  report  was  available),  and  data  on  the  dollar  value  of  SBIR 
awards  by  participating  agencies  in  fiscal  year  2009  (the  latest  year  for 
which  SBA  could  provide  the  data).  We  examined  relevant  documents 
from  participating  agencies  for  fiscal  years  2008  through  2010,  and  for 
fiscal  year  201 1  when  possible.  Documents  we  reviewed  included 
solicitations  for  applications  issued  by  each  of  these  agencies, 


^IH  accounted  for  more  than  98  percent  of  the  SBIR  expenditures  of  the  Department  of 
Health  and  Human  Services  from  fiscal  year  2000  through  fiscal  year  2010.  Four  other 
agencies  in  the  department — the  Centers  for  Medicaid  and  Medicaid  Services,  the 
Centers  for  Disease  Control  and  Prevention,  the  Administration  for  Children  and  Families, 
and  the  Food  and  Drug  Administration — also  issued  SBIR  solicitations  during  that  period. 


Page  30 


GAO-1 1-698  Small  Business  Innovation  Research 


Appendix  I:  Scope  and  Methodology 


instructions  to  applicants,  minutes  from  meetings  of  SBA  and  SBIR 
program  directors,  performance  plans  and  reports,  descriptions  of 
commercialization  assistance  provided  to  SBIR  awardees,  and  minutes 
from  meetings  of  agency  advisory  committees.  In  addition,  we  also 
identified  and  interviewed  SBIR  program  officials  at  each  agency  and 
officials  responsible  for  implementing  programmatic  goals.  For  these 
interviews,  we  asked  a  standard  set  of  questions  to  help  ensure  that  we 
obtained  consistent  information  about  the  SBIR  programs  at  each  of  the 
agencies.  We  also  interviewed  inspector  general  staff  at  NSF,  which 
facilitated  SBIR-related  activities  conducted  by  the  Council  of  Inspectors 
General  on  Integrity  and  Efficiency.  Finally,  we  interviewed 
representatives  of  trade  associations  about  their  views  of  the  SBIR 
program.  We  selected  the  trade  associations  on  the  basis  of  their 
familiarity  with  the  program,  the  technologies  on  which  they  focus,  and 
whether  their  membership  includes  small  businesses  owned  by 
disadvantaged  groups  and  women.  The  views  of  the  representatives  of 
these  associations  cannot  be  generalized  to  other  associations. 

To  determine  the  extent  of  SBIR  program  data  available  to  evaluate 
progress  in  increasing  commercialization  of  SBIR  technologies,  we 
reviewed  documents  related  to  SBA’s  SBIR  database,  including  terms  of 
work,  work  schedules,  and  proposed  guidance  related  to  the  development 
of  the  government-use  portion  of  the  database.  For  the  five  SBIR 
participating  agencies  whose  programs  we  reviewed,  we  examined 
documents  dating  from  2002  through  201 1 ;  these  documents  reflected 
commercialization  data  for  SBIR  award  recipients  that  had  received 
awards  from  1983  (the  first  year  in  which  agencies  issued  SBIR  awards) 
through  2010.  The  documents  we  reviewed  included  surveys  and  other 
data  collection  instruments  that  the  agencies  used  to  gather 
commercialization  information  from  award  recipients;  reports  on  data 
collection  results,  including  any  information  on  SBIR  award  spending 
during  the  years  corresponding  to  those  covered  in  each  of  the 
commercialization  data  collection  efforts;  and  anecdotal  descriptions  of 
commercialization  success.  We  also  reviewed  agency  solicitations  from 
fiscal  years  2008  through  201 0 — and  for  fiscal  year  201 1  when  possible — 
that  contained  reporting  requirements  for  award  recipients.  We 
interviewed  officials  at  SBA  and  each  of  the  five  participating  agencies 
included  in  our  review  to  obtain  information  on  the  specific 
commercialization  metrics  they  use  to  monitor  the  commercialization 
experience  of  award  recipients,  the  history  of  each  agency’s  data 
collection  efforts,  and  the  agencies’  experience  in  obtaining  such 
information  from  current  and  past  award  recipients. 
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We  conducted  this  performance  audit  from  June  2010  to  August  2011,  in 
accordance  with  generally  accepted  government  auditing  standards. 
Those  standards  require  that  we  plan  and  perform  the  audit  to  obtain 
sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for  our 
findings  and  conclusions  based  on  our  audit  objectives.  We  believe  that 
the  evidence  we  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 
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U  S  Small  Business  Administration 

Washington,  DC  20416 


.July  28,2011 

Mr  Frank  Rusco 

Director,  Natural  Resources  and  Environment 
Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr  Rusco: 

Thank  you  for  your  letter  dated  June  30, 201 1,  seeking  the  U.S.  Small  Business 
Administration’s  (SBA’s)  comment  on  the  US  Government  Accountability  Office 
(GAO)  draft  report  entitled:  “Small  Business  Innovation  Research:  SBA  Should  Work 
with  Agencies  to  Improve  the  Data  Available  for  Program  Evaluation.” 

We  do  not  have  any  issue  with  the  facts  and  findings  presented  in  the  draft  report.  In 
addition,  we  generally  agree  with  the  recommendations,  and  offer  the  following  plan  of 
action  in  response 

With  respect  to  the  first  recommendation:  “Collect  data  on  the  number  of  applications 
submitted  by  small  businesses  owned  by  disadvantaged  individuals  and  women,”  we  plan 
the  following  actions: 

•  The  SBA  currently  collects  the  number  of  awards  and  aggregate  amount  of 
awards  to  women,  minority-owned,  and  HUBZone  smalt  business  concerns  for 
the  SBA-issued  Annual  Report  to  Congress. 

•  In  the  new  SBA  database  at  SBIR.gov,  we  plan  on  collecting  non-awardee 
infoimation  at  an  appiicant-level  from  agencies  This  information  could  include 
designations  of  women-owned,  minority-owned,  and  HUBZone  small  business 
concerns.  Additionally,  this  information  on  non-awardees-  as  stated  in  the  report 
-  can  be  compared  with  information  in  the  Central  Contractor  Registration 
database  in  order  to  determine  whether  the  applicant  is  a  women-owned, 
minority-owned  or  HUBZone  small  business  concern  We  anticipate  collecting 
this  infoimation  in  FY2012,  to  allow  agencies  to  organize  their  databases  in  line 
with  our  new  data  collection  schema. 

•  This  fell,  the  SBA  plans  to  hold  a  workshop  to  share  best  pr  actices  in  outreach  to 
socially  and  economically-disadvantaged  groups  to  apply  to  the  SBIR  program 
The  outcome  of  the  workshop  should  be  a  commitment  from  agencies  to  baseline 
their  current  ievel  of  applicant  and  awardee  activity  Som  socially  and 
economically-disadvantaged  groups.  If  appropriate,  agencies  should  make 
commitments  to  obtain  applicant  numbers  from  these  groups. 


Page  33 


GAO-1 1-698  Small  Business  Innovation  Research 


Appendix  II:  Comments  from  the  Small 
Business  Administration 


With  respect  to  the  second  recommendation:  “Identify  best  practices  for  verifying  the  accuracy 
of  data  related  to  progress  in  increasing  commercialization,”  we  pian  the  following  actions: 

•  We  will  look  to  identify  Best  Practices  and  verification  methods  and  work  with  agencies 
toward  their  implementation 

*  While  out  plan  is  to  collect  commercialization  metrics,  the  recommendation  also 
addresses  the  tracking  of  such  information  to  determine  progress  in  “increasing 
commercialization”.  The  current  effort  for  data  collection  is  to  establish  a  baseline  of 
such  information  for  the  program.  Once  that  it  is  established,  we  can  review  the  progress 
of  increasing  commercialization. 

SBA  looks  forward  to  working  with  you  and  the  SBIR  participating  agencies 
If  you  have  any  questions,  please  contact  Shawn  McKeehan  at  202-205-7729. 

Sincgrely, 


gear  J.  Greene 

Associate  Administrator  for  Investment 
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Department  of  Energy 
Office  of  Science 
Washington.  DC  20585 

July  27,  2011 


Mr.  Franklin  Rusco 

Director,  Natural  Resources  and  Environment 
Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Rusco, 

Thank  you  for  the  opportunity  to  comment  on  the  draft  Government  Accountability  Office 
(GAO)  report  entitled,  “Small  Business  Innovation  Research:  SBA  Should  Work  with 
Agencies  to  Improve  the  Data  Available  for  Program  Evaluation”  (GAO- i  i -698).  We  have 
reviewed  the  draft  report  and  provide  general  comments  below.  The  comments  provided 
here  have  been  coordinated  with  other  relevant  offices  of  the  Department  of  Energy  (DOE). 

The  GAO  was  charged  by  the  Energy  and  Environment  and  the  Technology  and  Innovation 
Subcommittees  of  the  House  Committee  on  Science  and  Technology  “to  determine  (1)  how 
participating  agencies  have  addressed  the  SBIR  program’s  four  overarching  purposes  when 
implementing  their  programs  and  (2)  the  extent  of  SBIR  program  data  available  to  evaluate 
progress  in  increasing  commercialization  of  SBIR  technologies.”  We  recognize  that 
conducting  a  review  of  the  five-noted  SBIR  agencies  and  associated  activities  is  an  enormous 
undertaking,  and  in  particular,  we  appreciate  the  time  and  thoroughness  the  GAO  took  to 
review  the  Department’s  SBIR  program. 

With  regard  to  the  two  specific  recommendations  provided  by  GAO,  we  concur  with  these 
recommendations.  DOE  does  collect  data  on  the  number  of  applications  submitted  by  small 
businesses  owned  by  disadvantaged  individuals  and  women  and  is  willing  to  include  this  data 
in  its  reporting  to  SBA.  DOE  is  also  interested  in  learning  about  the  best  practices  for 
verifying  the  accuracy  of  data  related  to  progress  in  increasing  commercialization.  It  should 
be  noted  that  the  reason  DOE  does  not  currently  verily  commercialization  data  stems  from 
limitations  on  resources  and  not  from  a  belief  that  verification  is  of  limited  value. 

In  addition,  we  would  like  to  provide  one  comment  regarding  the  second  charge  to  GAO. 

The  GAO  report  discusses  the  variations  in  commercialization  data  collected  by  the  agencies 
and  the  challenges  that  result  when  frying  to  do  program-wide  assessment.  Implicit  in  this 
discussion  is  the  idea  that  there  is  one  set  of  metrics  that  can  be  used  to  compare  program 
success  across  all  participating  agencies.  Agencies  have  different  missions,  and  as  noted  in 
the  report,  also  include  non-financial  factors,  (e.g.  national  security)  in  assessing  their  SBIR 
programs.  The  GAO  report  does  not  address  the  underlying  questions  of  whether  (1)  there 
exist  a  universal  set  of  metrics  for  measuring  the  success  of  SBIR  programs  across  multiple 
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agencies  and  (2)  what  those  universal  metrics  should  be.  Until  these  questions  are  addressed, 
the  compatibility  of  the  available  data  among  agencies  will  remain  a  secondary'  concern. 

With  regard  to  the  draft  report’s  content,  please  find  attached  specific  comments  and 
suggested  edits  for  your  consideration. 

Thank  you,  again,  for  the  opportunity  to  provide  comments  on  this  draft  report,  if  you  have 
any  quesitons  or  concerns,  please  call  Manny  Oliver  at  (301)  903-0309.  We  look  forward  to 
receiving  your  final  report. 


Patricia  M.  Dehmer 

Deputy  Director  for  Science  Programs 

DOE  Office  of  Science 


Attachment 
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OFFICE  OF  THE 
DIRECTOR 


NATIONAL  SCIENCE  FOUNDATION 

4201  WILSON  BOULEVARD,  Room  1270 
ARLINGTON,  VIRGINIA  22230 
Tel.  703-292-8040  -  Fax.  703-292-9040 


July  29,  2011 


Mr.  Franklin  Rusco 

Director,  Natural  Resources  and  Environment 
United  States  Government  Accountability  Office 
Washington,  DC  20548 

Dear  Mr.  Rusco: 

The  National  Science  Foundation  (NSF)  appreciates  the  opportunity  to  review  and  comment  on  the 
Government  Accountability  Office's  (GAO's)  draft  report  entitled  Small  Business  Innovation  Research: 
SBA  Should  Work  with  Agencies  to  Improve  the  Data  Available  for  Program  Evaluation  (GAO-11-698). 

NSF  has  a  long  history  of  participation  in  the  Small  Business  Innovation  Research  (SBIR)  program  dating 
back  to  the  1980s  and  has  strongly  supported  the  program's  overarching  purposes  (i.e.,  to  meet  federal 
R&D  needs;  to  stimulate  technological  innovation;  to  increase  commercialization  of  federal  R&D-based 
innovations;  and  to  encourage  participation  by  businesses  owned  by  women  and  disadvantaged 
individuals).  NSF  has  been  proactive  in  addressing  the  program's  objectives  through: 

•  broad  topical  solicitations  that  emphasis  commercial  potential; 

•  mentorship  of  its  small  business  clients  by  program  directors  with  extensive  business  and 
technical  backgrounds; 

•  technical  assistance  in  commercialization  planning; 

•  matching  funds  to  stimulate  investment  in  innovative  projects; 

•  supplemental  awards  and  outreach  activities  to  encourage  members  of  underrepresented 
groups  to  participate  in  the  small  business  innovation  process;  and 

•  a  proactive  process  to  capture  the  commercial  outcome  of  NSF  Phase  II  awards  over  time. 

NSF  concurs  with  the  underlying  goals  of  the  GAO's  recommendations— to  enhance  the  collection  and 
analysis  of  SBIR  program  data — and  endorses  the  importance  of  sharing  best  practices  for  verifying  data 
accuracy.  NSF  is  committed  to  collaborate  with  the  Small  Business  Administration  and  the  other  SBIR- 
participating  agencies  in  the: 

•  implementation  of  a  government-use  data  base  for  program  evaluation  particular  focused  on 
commercial  outcomes; 

•  collection  of  data  on  the  participation  of  underrepresented  individuals  in  the  small  business 
innovation  process;  and 

•  identification  of  best  practices  for  verification  of  data  related  to  commercialization  outcomes. 
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Page  and  footnote 
numbers  in  the  draft 
report  may  differ  from 
those  in  this  report. 


NSF  commends  the  GAO  audit  team  for  an  outstanding  job  of  capturing  the  characteristics  and  features 
of  the  SBIR  process  at  NSF,  as  reflected  in  the  GAO-developed  statement  of  facts  and  in  this  draft  report. 
In  reviewing  the  draft  report,  a  few  inaccuracies  were  identified;  therefore,  the  following  amendments 
to  the  draft  report  are  recommended. 

Page  7,  Footnote  17.  The  report  on  the  NRC  Assessment  of  the  SBIR  Program  at  NSF,  published  in  200S, 
should  be  noted  {consistent  with  similar  reports  for  other  agencies  that  are  cited  in  the  footnote). 
Propose  the  following,  "...prepublication  in  November  2007,  the  DOE  and  NASA  reports  were  released  in 
prepublication  in  June  2008  and  December  2008,  respectively,  and  the  NSF  report  was  issued  in  2008." 

Page  23,  Line  8.  The  provided  dollar  figure  of  "$1.5  billion"  that  appears  in  the  draft  is  incorrect.  It 
should  read..."evaluated  from  July  2005  through  May  2010  had  realized  a  total  of  $1.05  billion  in 
commercial  revenue." 

Page  23,  Footnote  39.  As  written,  the  footnote  introduces  several  inaccuracies.  A  revised  version  is 
offered.  "NSF's  broad  estimate  of  commercial  revenue  includes  sales  revenue  from  products  and/or 
services  that  are  a  consequence  of  the  award,  licensing  revenue,  revenue  from  mergers  or  sales  of 
award  recipients'  businesses  that  resulted,  at  least  in  part  from  the  awards.  NSF's  estimate  does  not 
include  venture  capital  funding.” 

Thank  you  for  the  opportunity  to  comment  on  this  draft  report.  If  you  have  any  questions  regarding  this 
response,  please  contact  Kathryn  Sullivan  at  703-292-7375.  We  look  forward  to  receiving  your  final 
report. 


Sincerely, 


Subra  Suresh 
Director 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday 
afternoon,  GAO  posts  on  its  Web  site  newly  released  reports,  testimony, 
and  correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted 
products,  go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 
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Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 
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Washington,  DC  20548 
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